The present invention relates to the analysis of real-time streams of time-series data.
Computer systems and applications are growing more complex and logging increasing amounts of data to allow for diagnostics and maintenance. The analysis of this data to find patterns and relationships grows exponentially more expensive as the size and complexity of the systems increase. For real-time systems, it may be desirable to analyze this data as fast as it is being collected, and with minimal delay.
It is often desirable to analyze large sets of time-series data, that is, data consisting of sets of values with associated times. Real-time analysis of time-series data as it is received is referred to as “streaming analytics.” Such data may be generated, for example, by monitoring services for IT systems in which the values are details of error messages and the like, and telecommunications systems. A stream of data will have associated with it one or more “metrics,” which are values that can be derived from the data stream. It may be desirable to perform correlation analyses of the data stream, referred to as “streaming correlation analysis,” to determine if different metrics associated with the stream are correlated. In particular, for example, when the current metrics of two sets of time-series data now indicate a low level of correlation, when previously the metrics of the two sets of time-series data indicated a high level of correlation, this can be an anomalous situation that may indicate a problem.
The hardware requirements for such streaming analytics and streaming correlation analysis, using typical analytic techniques and approaches, requires memory and processing power that is far greater than that required to collect the data. A system that will identify patterns and relationships in streaming data, such that system operators can easily consume and act upon the information, would be advantageous. Further, it would be advantageous if such a system were implemented with a small memory and processing footprint.